Today's audiences in motion picture theaters are presented with more imposing images throughout a motion picture than they had been given in the past. The “big screen” was a part of the movie-going experience of the past, and there was a trend for several decades to shrink the screen. That way, the design of a movie complex could incorporate multiple screens, rather than the single screen that was customary during the “classic film” era. Today, the concept of the “big screen” is coming back and taking on a new meaning, as screens become larger and new installations include screens that are 90 feet or 27 meters wide. With a fixed number of pixels across the width of such a large screen, the amount of motion displacement of each pixel from one frame to the next increases proportionately with the size of the screen. For example, in the 2K format, each frame of a motion picture would contain 2048 pixels in each horizontal line, from the left edge to the right edge of the screen. With 1080 inches across a 90-foot screen, there is 0.527 inches per pixel of motion displacement between frames, even for a pixel that appears to move only one position. That is more than half of one inch, equivalent to 1.34 cm. Even with the 4K format, which provides for 4096 pixels on each horizontal line, there is still more than one-quarter of an inch or nearly seven-tenths of a centimeter of motion displacement between frames for any given pixel, at a minimum.
At the traditional frame rate of 24 frames-per-second, this amount of motion displacement had created an undesirable artifact for viewers who watch motion pictures on such a large screen. For newly-produced motion pictures, doubling the frame rate to 48 frames-per-second is a groundbreaking practice that could become a new industry standard, because it mitigates this undesirable artifact, as well as others associated with a low frame rate. To contemporary motion picture audiences, the traditional frame rate of 24 frames-per-second is rapidly becoming obsolete, especially for today's motion pictures, which feature dramatic action scenes. Even ultra-large-screen systems like film-based IMAX retain such artifacts as judder, stepping, strobing, and the appearance of image instability that are inherent in 24-frame-per-second display, when that frame rate is used for both photography and exhibition.
While high frame rates have been used mostly for special-venue presentations until now, the industry is beginning to notice the potential of such frame rates for general-release motion pictures. Peter Jackson's 2012 production of The Hobbit: An Unexpected Journey was released in a version photographed at 48 frames-per-second. The result was a dramatic increase in perceived realism, compared to presentation at 24 frames-per-second; perhaps an excessive increase in perceived realism. There is no need to recount the advantages of high-frame-rate presentation here. In the late 1950s, producer Michael Todd photographed Oklahoma! and Around the World in Eighty Days in the conventional 24-fps format, and also in Todd-AO, a format that featured photography and exhibition at thirty frames-per-second. In the early 1990s, Weisgerber taught a means for transitioning between 24 and 30 frames-per-second by increasing underexposure of selected sequences of frames to deliver increased flicker during those sequences, thereby producing a sharp transition between sequences photographed and exhibited at 24 frames-per-second, and those photographed and exhibited at 30 frames-per-second, in U.S. Pat. No. 5,096,286.
Later, Weisgerber taught the advantages of high frame rates in his early film-oriented inventions. U.S. Pat. No. 5,627,614 taught the selective use of 48 frames-per-second for some parts of a motion picture, along with the traditional frame rate of 24 frames-per-second for the other parts of the same motion picture. He also taught the same advantages for other frame-rate combinations in U.S. Pat. No. 5,793,894. It should be noted that the previous Weisgerber inventions highlighted the difference in appearance between the “cinematic” look of 24-frame-per-second photography and presentation, and the improved, high-impact look available through 48-frame-per-second photography and presentation. The same result can also be obtained with other frame rates. These inventions were not suitable for previously-produced motion pictures, but worked only for new productions.
More recently, Weisgerber taught the use of a compact film format for photography, along with digital projection at 48 frames-per-second in application Ser. No. 11/796,411, which has been allowed, and on which a patent will soon be issued. As with Jackson's Hobbit, that invention requires that the original motion picture be photographed and displayed at 48 frames-per-second, and it applies only to new motion pictures. For previously-produced motion pictures, nearly all of which were photographed or otherwise produced at 24 frames-per-second, an upgrade to the higher frame rate of 48 frames-per-second is required for exhibition at that speed. With 48 frames-per-second becoming a new standard for theatrical motion picture display, thousands of previously-produced motion pictures run the risk of being relegated to the dustbin of history; satisfying only “film buffs” and nostalgia-seekers, for whom the artifacts inherent in 24-frame-per-second photography and exhibition are part of the experience of watching a “classic” film. For everyone else, essentially all previously-produced 24-frame-per-second motion pictures would appear antiquated, and contemporary audiences might avoid them for that reason.
It is, therefore, an important object of this invention to give new audiences an opportunity to view previously-produced motion pictures at a level of image quality to which they are becoming accustomed and, by so doing, generate new revenue streams for their owners in a manner that has not been feasible before. This is feasible now, because of a method for correcting a specific artifact imparted during the conversion of 24-frame-per-second motion pictures for display at 48 frames-per-second. This artifact is caused by repeating entire frames, when they depict complex or chaotic motion. This deficiency is left uncorrected as in the prior art, and it would detract significantly from the quality of presentation to contemporary audiences.
It is envisioned that a significant use of this invention will be converting motion pictures originally photographed at 24 frames-per-second, for enhanced presentation at 48 frames-per-second. This can apply to classic motion picture films, when the images on the film frames have been converted to digital form by means known in the art. In addition, the invention can also be used for other applications in motion picture production. Stock footage originally photographed at 24 frames-per-second can be upgraded for use in new motion pictures photographed at 48 frames-per-second. The same holds true for CGI images captured or otherwise produced at 24 frames-per-second and converted to 48 frames-per-second for eventual theatrical exhibition. Any application that involves the method described should be considered as lying within the scope of the invention.
Today's audiences are accustomed to receiving more visual information than audiences had absorbed in the past. HDTV scans sixty fields per second, equivalent to a 60-frame-per-second motion picture like those photographed and exhibited in Trumbull's Showscan system. Showscan succeeded, in large part, because it was an immersive system that featured a high frame rate and a large screen. It was used primarily for special-venue films, where the experience was often like an amusement ride. There was no need for the sort of cinematic art and special effects that are necessary components of today's motion pictures that are produced for theatrical exhibition.
In short, any means for converting a motion picture photographed at a low frame rate for exhibition at a high frame rate must do so without sacrificing any image quality, from the standpoint of the quality of presentation that the members of the audience see. While there are software-based inventions that can change frame rates for slow-motion special effects, they cannot reliably produce corruption-free images that are suitable for high-frame-rate presentation.
Hazra, U.S. Pat. No. 5,693,614 teaches a method for color-correction of images when changing presentation rates. His method is drawn to computer screens, so it would not apply to large-screen theatrical motion picture presentation, especially on today's largest screens. The amount of motion displacement of each pixel on a 90-foot screen is 49 times the amount of motion displacement on a 22-inch computer monitor screen. Another weakness in Hazra's method is that it calls for entire frames to be repeated if a frame is found to be corrupted, and not suitable for treatment according to the method claimed in that invention. This may not pose a problem of perception for the viewer of a computer screen, but it would cause an objectionable artifact for a viewer sitting in a theater and watching a motion picture on a large screen.
Other inventors such as Cok have advanced digital image storage and mixing of frame rates for sequences of digital images in inventions such as U.S. Pat. No. 7,242,850, but Cok is also silent on a specific method for correcting corrupted images, short of repeating the entire preceding image. Cok claims image interpolation as part of a method for improving image quality by increasing the frame rate (claims 4 and 25), but he treats the “frame” as an undivided entity throughout his specification and claims. He never treats some pixels in a specific image differently from other pixels in the same image. In a related invention (US20020149696), Cok teaches interpolation of images obtained through motion vector analysis. However, he goes on to say (at ¶16, lines 6-7): “The frame rate of portions of the image sequence which are not interpolated may be increased by simple frame replication.” In other words, Cok only replicates entire frames, as does Hazra 614, when “Portions of the image sequence that will not improve noticeably or which will include objectionable artifacts resulting from the interpolation, may be excluded from interpolation” (Cok, ¶16 at lines 3-4). The present invention eliminates this artifact that would degrade a theatrical presentation under the Hazra and Cok methods.
This is important to contemporary audiences, who perceive a large image as being part of the movie-going experience, and who have become accustomed to the level of picture dominance available with such large-screen systems as IMAX (15-perforation film format). IMAX is generally limited to special venues, where the architecture of the theater is specifically designed to accommodate the size of the screen, which extends from wall to wall and from floor to ceiling. While there have been experiments with a 48-frame-per-second version of IMAX, the preferred embodiment of that system, at least at the present time, calls for photography and exhibition at 24 frames-per-second of motion pictures specially-produced according to it. Despite the large image size, these motion pictures contain the undesirable artifacts of the low frame rate, including judder, stepping, strobing and noticeable motion displacement between frames.
However effective the methods taught by Cok and others may be at synthesizing images for insertion between successive pairs of images from an existing motion picture, there are limits to such effectiveness. Methods such as Cok's may be able to produce “in-between” images that will resemble the appearance of live action that had actually been photographed at double the rate of image-capture in real time, but these methods are not capable of dealing with images that have become corrupted. With a corrupted image, it is necessary under the prior art to repeat that corrupted frame in its entirety. If a motion picture that has been upgraded for display at 48 frames-per-second contains shots where entire frames are repeated as they were originally photographed at 24 frames-per-second, the motion picture will appear to stutter at those points. In other words, the motion will not appear smooth. A smooth appearance is the object of any upgrade for a previously-produced motion picture. Weisgerber 614 demonstrated an understanding of the difference between the appearance of a motion picture at 48 frames-per-second, as opposed to 24 frames-per-second. In that invention, he captured and displayed certain scenes or sequences at 48 frames-per-second to produce a high-impact impression on the audiences watching motion picture films. The other scenes or sequences were photographed and displayed at the conventional film frame rate of 24 frames-per-second (actually, those frames were double-frame printed and the entire motion picture was projected at 48 frames-per-second). These portions of the motion picture delivered a “cinematic” appearance typically associated with film exhibited at the lower frame rate.
If a motion picture has been enhanced for high-frame-rate (HFR) presentation, any repetition of entire frames; even a single frame, would distract the viewers and preclude the desired effect, at the time those specific frames are repeated. The repetition of even a single frame would create an undesirable artifact that audiences accustomed to HFR exhibition would notice, and would consider unappealing. In effect, it would appear to an audience that the frame in question was frozen in time; its motion would not appear real. The repeated frames would have this freeze-frame appearance, which would be jarring and detract from the desired HFR appearance. In fact, such a result contradicts the object of high-frame-rate (HFR) presentation, which is to maximize the fluidity and consistency of motion; to augment the appearance of realism of a motion picture, as the audience views it.
A fundamental objective of HFR 48-frame presentation concept is to mitigate the judder and motion artifacts that are present in 24-frame-per-second motion pictures. Then any repeated frames, such as Cok's patent 850 incorporates, will be defective and useless in the 48-frame-per-second cinema method described in the present invention. The up-conversion of 24-fps to 48-fps must incorporate true, smooth frame-to-frame depiction. Otherwise, the viewer will perceive a disturbance in motion, especially in action sequences, or those with significant motion or camera movement.
Because of this, any repetition of entire frames is completely impractical, and incompatible with HFR exhibition practice. That is because displaying a motion picture with some frames repeated cannot deliver the smooth, artifact-free presentation which is compatible with 48-fps exhibition. In short, the repetition of even a single frame constitutes a failure to meet contemporary presentation standards. While there are certain pixels in the images of motion pictures that cannot be subjected to Cok's method and placed accurately in a synthesized “in-between” image, Cok's method cannot salvage the rest of the image, which often comprises most of it. Instead, his method repeats entire images, which fails to meet today's HFR presentation requirements.
Therefore, it is a further object of this invention to minimize the amount of repetition of pixels by selecting only corrupted pixels from affected images, and repeating only the specific pixels that are corrupted, rather than the entire previous frame. In that manner, most of the pixels in the corrupted frames will be treated in the same manner as all pixels in uncorrupted frames, with new images synthesized and inserted between each image from the original motion picture and its successor. In the present invention, only the corrupted pixels, which comprise only a small percentage of the totality of each corrupted image, would be repeated. The amount of pixel repetition will be so small as to be unnoticed by the audiences, who will perceive an entire motion picture presentation as a “seamless” one, with entirely smooth motion.